Likelihood Ratio Gradient Estimation for Steady-State Parameters

Abstract: We consider a discrete-time Markov chain Φ on a general state-space X, whose transition probabilities are parameterized by a real-valued vector θ. Under the assumption that Φ is geometrically ergodic with corresponding stationary distribution π(θ), we are interested in estimating the gradient ∇α(θ) of the steady-state expectation α(θ) = π(θ)f. To this end, we first give sufficient conditions for the differentiability of α(θ) and for the calculation of its gradient via a sequence of finite horizon expectations. We then propose two different likelihood ratio estimators and analyze their limiting behavior.